Abstract: Training a very deep neural network is a challenging task, as the deeper a neural network is, the more non-linear it is. We compare the performances of various preconditioned Langevin algorithms with their non-Langevin counterparts for the training of neural networks of increasing depth. For shallow neural networks, Langevin algorithms do not lead to any improvement, however the deeper the network is and the greater are the gains provided by Langevin algorithms. Adding noise to the gradient descent allows to escape from local traps, which are more frequent for very deep neural networks. Following this heuristic we introduce a new Langevin algorithm called Layer Langevin, which consists in adding Langevin noise only to the weights associated to the deepest layers.

We have a paradox staring us in the face. All that web content creates a great forum for philosophical debate: Will AI save the world or bring about the extinction of homo sapiens? Compelling research demos often show super-human performance on selected cognitive tasks, especially perception and pattern recognition in image, streams, audio, and transaction data. Thus, the possible outcomes also include affordable solutions for complex social problems, advanced diagnosis and treatment of medical conditions, environmental sustainability efforts, climate optimized traffic flow and safety, and cybercrime and fraud prevention. And of course, businesses everywhere appear to be fully expecting to put AI to work in some form as soon as possible.